On Generalization by Neural Networks

We report new results on the corner classification approach to training feedforward neural networks. It is shown that a prescriptive learning procedure where the weights are simply read off based on the training data can provide good generalization. The paper also deals with the relations between the number of separable regions and the size of the training set for a binary data network. Prescriptive learning can be particularly valuable for real-time applications. © 1998 Elsevier Science Inc. All rights reserved. I . I n t r o d u c t i o n A new approach to training feedforward neural networks for binary data was proposed by the author [1,2]. This is based on a new architecture that depends on the nature of the data. It was shown that this approach is much faster than backpropagation and provides good generalization. This approach, which is an example of prescriptive learning, trains the network by isolating the corner in the n-dimensional cube of the inputs represented by the input vector being learnt. Several algorithms to train the new feedforward network were presented. These algorithms were of three kinds. In the first of these (CC1) the weights were obtained upon the use of the perceptron algorithm. In the second (CC2), the weights were obtained by inspection from the data, but this did not provide generalization. In the third (CC3), the weights obtained by the second method were modified in a variety of ways that amounted to randomization and which now provided generalization. During such randomization some I E-mail: [email protected] 0020-0255/98/$19.00 © 1998 Elsevier Science Inc. All rights reserved. PII: S 0 0 2 0 0 2 5 5 ( 9 8 ) 1 0009-9 294 S.C. Kak / Information Sciences 111 (1998) 293-302 of the learnt patterns could be misclassified; further checking and adjustment of the weights was, therefore, necessitated. Various comparisons were reported in [3-5]. The comparisons showed that the new technique could be 200 times faster than the fastest version of the backpropagation algorithm with excellent generalization performance. In this article we show how generalization can be obtained for such binary networks just by inspection. We present a modification to the second method so that it does provide generalization. This technique's generalization might not be as good as when further adjustments are made, but the loss in performance could, in certain situations, be more than compensated by the advantage accruing from the instantaneous training which makes it possible to have as large a network as one pleases. Experimental results in support of our method are presented.